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EXECUTIVE  SUMMARY 


Following  considerable  criticism,  changes  in  the  Operational  Test  and 
Evaluation  (OT&E)  policy  were  established  by  Mr.  David  Packard  in  1971. 

This  created  a new  test  term  called  Initial  Operational  Test  and  Evalua- 
tion (IOT&E).  IOT&E  is  often  performed  by  the  user  prior  to  system  develop- 
ment phase  completion.  The  user  report  is  used  as  one  of  the  primary  factors 
in  determining  a system's  future  fate.  In  the  case  of  major  programs,  it 
is  a requirement  prior  to  the  Defense  Systems  Acquisition  Review  Council 
(DSARC)  conducted  prior  to  a production  decision.  Due  to  development  pro- 
gram schedules  and  funding,  most  IOT&E  evaluations  are  conducted  simultane- 
ously with  the  Development  Test  and  Evaluation  (DT&E).  In  this  situation, 
we  have  two  commands  working  together  in  a joint  DT&E/IOT&E  effort,  with 
the  overall  program  control  remaining  with  the  developing  command.  This 
report  concentrates  on  implementation  aspects  of  recent  joint  DT5E/I0TSE 
programs  conducted  by  the  Air  Force  Systems  Command  (ACSC)  and  the  Tactical 
Air  Command  (TAC). 

The  primary  purpose  was  to  gain  some  insight  into  how  well  the  combined 
testing  concept  is  working  out.  The  study  did  not  concentrate  on  any  one 
type  or  dollar  value  program,  but  instead  looks  across-the-board  at  the 
effectiveness  of  all  recent  TAC  IOT&E  efforts. 

Data  was  collected  by  both  personal  interviews  and  questionnaires. 
Interviews  were  conducted  within  the  Directorate  of  Operations,  Deputy 
Chief  of  Staff,  USAF.  Questionnaires  were  sent  to  both  AFSC  DT&E  program 
managers  and  TAC  IOT&E  project  managers  who  possess  experience  with  com- 
bined DT&E/IOT&E.  Since  the  Tactical  Air  Warfare  Center  (TAWC)  is  TAC's 
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primary  OT&E  test  agency,  TAC  questionnaires  were  sent  to  individuals  in 
this  center. 

Results  of  both  interviews  and  questionnaires  revealed  a definite 
positive  attitude  toward  the  combined  testing  concept.  In  addition,  inter- 
views were  most  helpful  in  providing  background  and  rationale  for  the 
creation  of  the  new  OT&E  group  called  Air  Force  Test  and  Evaluation  Center 
(AFTEC).  It  was  learned  that  the  establishment  of  AFTEC  had  no  relation  to 
past  performance  of  TAC  in  accomplishing  effective  IOT&E. 

Questionnaire  responses  were  compiled  and  analyzed  statistically 
first  to  determine  if  a group  opinion  to  a particular  question  was  present. 
Another  statistical  test  compared  the  mean  responses  from  questions  common 
to  both  AFSC  and  TAWC.  It  was  interesting  to  note  that  of  all  questions 
which  .xhibited  group  opinions,  the  opinion  expressed  was  toward  a positive 
rating  of  IOT&E  effectiveness. 

It  was  concluded  that  the  improved  two-way  exchange  between  AFSC  and 
TAWC  (TAC)  afforded  by  the  combined  DT&E/IOT&E  has  been  widely  accepted. 

It  was  further  concluded  that  TAC/TAWC  IOT&E  performance  has  been  excellent. 

It  is  recommended  that  more  emphasis  be  placed  on  combined  DT&E/IOT&E 
testing  to  resolve  earlier  the  differences  between  user  and  developer.  It 
is  also  recommended  that  the  using  command  designate  a project  monitor  very 
early  in  the  acquisition  cycle  who  will  ultimately  become  the  IOT&E  or  OT&E 
manager.  This  would  motivate  the  user/tester  to  influence  the  design  while 
still  in  a flexible  state. 
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A LOOK  INTO  THE  EFFECTIVENESS  OF  TAC  IOT&E 


A.  INTRODUCTION 

1.  Objective 

The  overall  purpose  of  this  effort  was  to  provide  the  author 
some  insight  into  the  effectiveness  of  the  Air  Force's  relatively  new 
Initial  Operational  Test  and  Evaluation  (IOT&E)  procedures. 

2.  Scope 

Past  studies  have  adequately  investigated  the  background  and 
rationale  leading  up  to  the  current  Department  of  Defense  (DOD)  policies 
on  IOT&E  (1,  2,  3).  These  test  philosophies  originally  instituted  in 
1970  by  the  then  Deputy  Secretary  of  Defense,  Mr.  David  Packard,  have 
been  re-emphasized  by  their  appearance  in  the  Air  Force  Regulation  23-35, 

Air  Force  Test  and  Evaluation  Center  (AFTEC),  1 January  1974,  and  in  the 
DOD  Program  of  Research,  Development,  Test  and  Evaluation  FY  75,  April 
1974,  and  by  a recent  April  1974  address  by  Deputy  Secretary  of  Defense 
Clements  to  the  Defense  Systems  Management  School  faculty  and  students 
(4,  5).  Therefore,  this  report  will  not  reiterate  in  detail  these  manage- 
ment philosophies,  but  instead,  will  introspect  some  of  the  implementation 
aspects  of  this  policy  as  they  apply  specifically  to  Tactical  Air  Command 
(TAC)  IOT&E.  A comprehensive  study  which  would  include  all  known  aspects 
relating  to  the  effectiveness  of  operational  testing  would  require  resources 
far  exceeding  those  available  under  the  author's  academic  restraints. 

Therefore,  I elected  to  investigate  only  aspects  of  TAC  IOT&E  testing 

which  are  conducted  concurrently  during  Development  Test  and  Evaluation  (DT&E). 
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3.  IOT&E  Defined 

IOT&E  is  that  Operational  Test  and  Evaluation  conducted  prior  to 
a production  decision.  The  intent  of  IOT&E  is  to  determine  factors  such 
as  operational  effectiveness,  suitability,  and  supportability  as  early  in 
the  systems  acquisition  cycle  as  possible  (6:6).  IOT&E  is  accomplished  by 
supporting  or  operating  commands  and  will  be  performed  on  pilot  production 
items  when  available.  In  the  case  of  major  systems  (i.e.,  those  with  develop 
ment  costs  in  excess  of  $200  million),  the  IOT&E  is  required  prior  to  the 
Defense  Systems  Acquisition  Review  Council  action  preceding  the  production 
decision  (normally  DSARC  III).  (7)  IOT&E  was  instituted  in  1971  by  the 
then  Deputy  Secretary  of  Defense,  Mr.  David  Packard.  His  memoranda  of 
11  February,  21  April,  and  3 August  1971,  which  were  later  superseded  by 
Department  of  Defense  Directive  5000.3,  "Test  and  Evaluation,"  19  January 
1973,  all  provided  basic  policies  for  IOT&E  (8,  9,  10).  The  memoranda 
were  results  of  Mr.  Packard's  own  investigation  into  Test  and  Evaluation 
activities,  results  of  the  Blue  Ribbon  Defense  Panel  (BRDP),  General  Ac- 
counting Office  reports,  and  criticism  from  Congress.  Some  of  the  primary 
criticisms  and  recommendations  were: 

a.  0T&E  was  poorly  designed  and  executed  and  was  generally  inadequate. 

b.  Recommended  that  D0D  establish  a Defense  Test  Agency  and  that  the 
Joint  Chiefs  of  Staff  (JCS)  perform  joint  operational  testing. 

c.  Recommended  that  the  military  Services  conduct  0T&E  by  an  organiza- 
tion independent  of  the  developer. 

Following  the  BRDP  report,  Mr.  Packard,  in  1971,  issued  three  memoranda 
on  0T&E  (8,  9,  10).  They  directed  that: 
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a.  OT&E  be  performed  by  an  agency  separate  and  distinct  from  the 
developer  which  evaluates  and  reports  the  test  results  directly  to  the 
Service  Chief. 

b.  IOT&E  be  performed  by  operational  personnel  in  as  realistic  an 
environment  as  possible  prior  to  the  production  decision. 

c.  Each  Service  would  establish  a staff  focal  point  for  OT&E  at  the 
headquarters  level . 

4.  Combined  DT&E/IOT&E  Defined: 

DT&E/ IOT&E  denotes  a testing  concept  whereby  the  IOT&E  is  con- 
ducted during  the  DT&E  phase.  This  is  normally  accomplished  when  program 
schedules,  costs,  or  production  lead  time  do  not  permit  independent  testing 
by  the  Operating  and  Supporting  Commands.  Although  the  system  is  under 
the  overall  control  of  the  Systems  Command,  the  Operating  Command  (user) 
conducts  a separate,  independent  evaluation.  The  results  of  this  IOT&E 
are  forwarded  through  separate  reporting  channels  to  HQ,  USAF.  (6) 

5.  Limitations: 

This  report  addresses  only  those  IOT&E  programs  conducted  con- 
currently during  the  Air  Force  Systems  Command's  (AFSC)  development  test 
and  evaluation  effort.  This  was  done  since  by  far  the  largest  majority  of 
all  current  IOT&E  testing  falls  within  this  category.  It  is  further  re- 
stricted to  only  joint  Development  Test  and  Evaluation  ( DT&E ) / I OT&E  pro- 
grams involving  the  AFSC  and  the  TAC,  respectively.  The  rigid  academic 
schedule  imposed  upon  the  author  limited  the  scope  of  this  effort  to  in- 
vestigating only  the  effectiveness  of  recent  combined  TAC  IOT&E.  Also, 
only  those  TAC  IOT&E  tests  conducted  prior  to  completion  of  full-scale 
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development  are  considered  (i.e.,  combined  DT&E/IOT&E).  No  attempt  was 
made  to  collect  information  on  any  specific  size  or  dollar  value  program. 
Therefore,  most  of  the  information  in  this  report  reflects  views  concern- 
ing programs  with  a dollar  value  less  than  major. 

6.  Organization  of  Report: 

Paragraph  B provides  information  on  the  methods  used  in  data  col- 
lection. It  also  includes  a general  description  of  the  analysis  made  and 
references  to  the  appropriate  annexes  for  details.  Paragraph  C discusses 
the  results  from  both  interviews  and  questionnaires.  The  statistical 
results  are  discussed  in  some  detail.  Paragraph  D provides  background  and 
mission  information  on  the  latest  OT&E  reorganization  in  the  Air  Force. 
Paragraph  E provides  conclusions  and  recommendations  resulting  from  this 
study. 
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B.  DATA  COLLECTION  AND  METHOD  OF  ANALYSIS 


1.  General: 

Data  for  this  report  was  collected  through  personal  interviews 
and  questionnaires.  The  questionnaires  were  used  to  provide  program  manag- 
er level  opinions  on  combined  DT&E/IOT&E  testing.  Two  slightly  different 
questionnaires  were  used.  One  was  tailored  for  the  AFSC  DT&E  program  manag- 
ers, and  one  for  the  TAC  IOT&E  project  managers.  The  majority  of  TAC's 
IOT&E  is  conducted  by  project  managers  in  the  Tactical  Air  Warfare  Center 
(TAWC).  Therefore,  for  TAC  information,  questionnaires  were  mailed  to 
TAWC.  When  TAWC  is  mentioned  from  time  to  time  in  this  report,  it  should 
be  remembered  that  this  is  TAC's  prime  OT&E  test  agency. 

2.  Report  Review: 

Initially,  one  intent  was  to  review  specific  recent  IOT&E  reports 
through  an  analysis  of  their  conclusions  and  recommendations  and  subsequent 
determination  of  HQ,  USAF  actions  resulting  from  these  reports.  This  sum- 
marization was  intended  to  indicate  to  what  degree  TAC  IOT&E  recommendations 
and  conclusions  were  being  followed.  The  rationale  was  that  this  would  pro- 
vide an  indication  of  the  credibility  of  the  reports  as  viewed  by  higher 
headquarters  and  in  turn  demonstrate  some  measure  of  IOT&E  effectiveness. 

It  was  decided  that  all  TAC  joint  DT&E/IOT&E  reports  published  during 
the  calendar  year  1973  would  be  used.  A listing  was  made  and  included 
the  TAWC  IOT&E  reports  listed  in  Table  1. 

My  efforts  to  accomplish  the  report  summation  task  were  beset  with 
problems  from  the  very  beginning.  A delay  in  obtaining  some  report  copies 
was  experienced  due  to  classification  and  several  were  never  received. 
Further,  the  recommendations  and  conclusions  varied  widely  in  content  and 


TABLE  1 


Recent  TAUC  IOT&E  Reports 


TITLE 

TEST  NO. 

DATE  PUBLISHED 

Compass  Robin 

73C-010U 

NOV 

1973 

IR  Proximity  Fuse 

73C-018T 

AUG 

1973 

TAC  IOT&E  of  Intermodel 
Modular  Container 

73C-024T 

SEP 

1973 

Airborne  Warning  and 
Control  System  (AUACS) 
Phase  I 

73C-045T 

JUL 

1973 

Evaluation  of  AN/PPS-15 
Surveillance  Radar 

73C-051T 

SEP 

1973 

Improved  ADAS,  A'l/ASQ-1 54 
(XA-2) 

73C-083T 

SEP 

1973 

IOT&E  of  Tactical  Air 
Cargo  Loader 

72C-014 

AUG 

1973 

Constant  Cone  AIM  7F 
OPEVAL 

72C-044 

MAR 

1973 

IOT&E  of  AN/PRC-104 
HF/SSB  Manpack  Radio 

72C-084 

JAN 

1973 

TAC  High  Line  Dock 

72C-130T 

JUN 

1973 

TAC  Information  Processing 
and  Interpretation  Segment 

72C-137T 

AUG 

1973 

I t 
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quantity  with  many  being  extensive  and  detailed.  For  example,  one  report 
contained  fourteen  recommendations  and  thirteen  conclusions.  It  became 
apparent  that  this  effort  would  be  both  difficult  and  time-consuming,  and 
in  addition,  would  have  produced  results  which  possibly  could  be  obtained 
through  personal  interviews  in  a much  shorter  time  period.  Therefore,  the 
report  review  was  abandoned. 
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3.  Interviews: 


Interviews  were  conducted  with  Headquarters,  United  States  Air 
Force  (HQ,  USAF)  personnel  in  the  Directorate  of  Operations;  specifically. 
Colonel  R.  W.  Phillips,  Deputy  Director  for  Operational  Test  and  Evalua- 
tion, and  two  other  personnel  in  his  division  were  interviewed.  These 
were  Lieutenant  Colonel  A.  W.  McRae,  Aircraft  and  Equipment  Test  Division, 
and  Mr.  H.  A.  Beck,  Test  Analysis  and  Evaluation  Division.  See  Organiza- 
tional Chart  (Figure  1). 

As  mentioned  earlier,  no  distinction  was  made  in  this  report  between 
major  and  minor  programs,  and  since  by  far  the  majority  fall  within 
the  latter  category,  this  led  me  to  Colonel  Phillips.  Had  I been  con- 
cerned primarily  with  major  programs  and  policy  implementation,  it  would 
have  undoubtedly  been  beneficial  to  interview  the  Deputy  Director  of  Test 
and  Evaluation  (Lieutenant  General  Starbird,  USA,  Ret)  within  the  Office 
of  the  Deputy  Director  of  Research  and  Engineering  (DDR&E),  or  some  of 
his  staff.  However,  I found  Colonel  Phillips  and  his  staff  to  be  very 
well  informed  on  the  latest  TAC  IOT&E  programs  and  so  cooperative  in  answer- 
ing my  questions  that  I felt  it  unnecessary  to  pursue  the  personal  inter- 
views further.  The  following  general  questions  are  of  the  type  posed  to 
Colonel  Phillips  and  his  staff: 

a.  Will  you  describe  your  organization  and  its  responsibilities 
as  related  to  Air  Force  IOT&E? 

b.  Specifically,  how  do  you  use  the  results  of  a typical  TAC 
IOT&E  report? 

c.  How  would  you  rate  the  quality  and  substance  of  recent  TAC 
IOT&E  reports  insofar  as  their  content  of  information  needed 
in  the  program  decision  process? 

d.  Can  you  describe  the  recent  Air  Force  OT&E  reorganization  and 
the  rationale  for  raking  the  change? 
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4.  Questionnaires: 


Questionnaires  were  mailed  to  both  AFSC  DT&E  program  managers  and 

TAWC  IOT&E  project  managers.  Twenty-one  questionnaires  were  mailed  to 

AFSC  program  managers  within  the  following  listed  Aeronautical  Systems 

Division  (ASD)  offices,  Wright-Patterson  AFB,  Ohio: 

RWD 

RWR 

RWE 

RWZ 

RWS 

RW86 

RD4T 

SOUP 

SD5T 


Twenty-one  slightly  different  questionnaires  were  mailed  to  TAC  IOT&E  pro- 
ject managers  in  the  TAWC,  Eglin  AFB,  Florida.  These  questionnaires  were 
mailed  to  personnel  in  the  following  listed  TAWC  offices,  all  of  which  are 
organizationally  under  the  Deputy  for  Test  and  Evaluation: 


Directorate  of  Reconnaissance 

(also  includes  TAC  drone  testing)  TER 

Directorate  of  Electronic  Warfare  TEW 

Directorate  of  Fighter  Test  & Evaluation  TEF 

Directorate  of  Command  & Control  TEO 

Directorate  of  Airlift  & Support  Opns  TEL 


An  example  of  each  questionnaire  type  is  included  as  an  annex  to  this 
report.  (See  Annex  A for  AFSC  questionnaire,  and  Annex  B for  TAWC 
questionnaire. ) 
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5.  Data  Analysis: 

All  information  collected  for  this  report  reflects  the  opinions 
and  judgments  of  individuals.  The  manner  in  which  it  is  presented  is  the 
personal  preference  of  the  author  resulting  from  his  particular  background 
and  experience.  Specific  information  collected  during  the  personal  inter- 
views was  recorded  through  conventional  note-taking  techniques.  The  question- 
naires were  designed  to  avoid,  wherever  possible,  simple  yes/no  answers 
by  providing  individuals  a range  within  which  to  express  their  opinions. 

They  were  also  given  an  opportunity  to  comment  on  each  question  if  they  so 
desired.  Fifteen  of  the  21  questionnaires  mailed  to  AFSC  were  returned, 
while  17  of  the  21  sent  to  TAWC  were  returned.  This  provided  an  overall 
response  of  76  percent.  For  data  analysis  and  clarification,  questions 
common  to  both  AFSC  and  TAWC  questionnaires  are  grouped  with  responses 
and  comments  from  each  source.  Questions  pertaining  to  only  one  question- 
naire type  are  each  listed  separately  with  corresponding  responses  and 
comments  (see  Annex  C).  Responses  are  plotted  as  bar-graphs  to  facilitate 
understanding  and  to  indicate  trends. 

Statistics  were  applied  to  responses  from  each  question  to  determine 
if  a group  opinion  for  that  question  was  actually  exhibited.  Due  to  the 
relatively  small  sample  size,  the  method  used  was  the  Kolmogorov-Smirnov 
one-sample  test  (11:300).  Once  it  was  established  that  a group  opinion 
for  a specific  question  did  exist,  the  mean  of  the  responses  from  the 
AFSC  group  was  compared  to  the  mean  of  the  responses  from  the  TAWC  group 
to  determine  if  any  significant  differences  existed  (12:233).  For  the 
reader  who  desires  to  follow  the  method  of  calculation,  an  example  of  each 


11 


type  is  included  in  Annex  D.  Each  questionnaire  contained  several  questions 
designed  to  indicate  only  whether  or  not  the  respondent  had  the  qualifi- 
cations or  experience  desired  for  answering  the  scaled  questions.  The 
qualification  questions  were  not  of  any  other  value  and  are  summarized  in 
Annex  C only  for  completeness. 
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C.  RESULTS  AND  DISCUSSION 


1.  Purpose: 

This  section  includes  the  analysis  of  all  information  collected 
during  this  effort.  I have  attempted  to  keep  the  results  as  objective  as 
possible,  and  therefore  relied  quite  heavily  on  statistical  analysis  of 
group  sample  responses. 

2.  Interviews: 

Prior  to  initiating  this  investigation  of  combined  DT&E/IOT&E,  I 
must  admit  that  some  suspicions  were  present  within  my  mind  that  all  was  not 
working  out  as  well  as  had  been  originally  intended.  I wanted  to  determine 
where  the  major  criticisms,  if  any,  would  lead  me.  I felt  that  this  could 
be  best  approached  through  interviews  with  the  proper  people  in  HQ,  USAF. 
First,  I desired  general  comments  or  feelings  on  the  adequacy  of  TAWC  in 
performing  its  role  as  TAC's  primary  test  organization  for  OT&E.  I in- 
tentionally narrowed  this  down  to  efforts  of  combined  DT&E/IOT&E,  since  by 
far  the  largest  percentage  of  tests  now  fall  in  this  category.  These  ob- 
jectives led  me  to  Colonel  Phillips  in  HQ,  USAF/XOOW.  Comments  from  Colonel 
Phillips  and  his  staff  were  most  encouraging.  It  was  learned  that  almost 
without  exception,  the  I0T&E  performed  by  TAWC  was  adequate  to  enable  any 
necessary  HQ,  USAF  program  decisions. 

The  recent  creation  of  the  Air  Force  Test  and  Evaluation  Center 
(AFTEC)  in  January  1974  raised  some  questions  as  to  the  background 
leading  up  to  this  decision.  I especially  wanted  to  determine  if  any  past 
problems  with  I0T&E  in  TAWC  had  any  tie-in  with  the  establishment  of  the 
new  test  center.  Mr.  K.  A.  Beck  of  Colonel  Phillips'  staff  had  extensive 
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experience  in  helping  to  organize  and  establish  AFTEC.  My  discussions 
with  Mr.  Beck  were  most  helpful,  as  they  provided  me  with  some  detailed 
information  on  the  background  and  operation  of  AFTEC  which  I had  not  pre- 
viously been  privileged  to  receive.  It  was  confirmed  that  AFTEC  establish- 
ment had  no  relationship  to  recent  TAC  I0T&E  performance.  Background  infor- 
mation leading  up  to  establishment  of  AFTEC  and  a brief  description  of  its 
mission  are  included  in  Section  D. 

3.  Questionnaires: 

The  opinions  expressed  by  Colonel  Phillips  and  his  staff  on  the 
excellent  quality  and  effectiveness  of  TAC  IOT&E  are  substantiated  by 
data  received  through  the  questionnaires.  It  is  interesting  to  note  that 
of  the  seven  questions  common  to  both  AFSC  and  TAWC  questionnaires,  only 
three  fell  into  the  category  possessing  group  opinions  from  both  sources. 

Two  of  these  questions  were  related  to  the  effectiveness  of  TAC  IOT&E.  The 
first  set  of  nuestions  requested  an  opinion  on  the  degree  to  which  the 
IOT&E  recommendations  have  been  or  will  be  followed.  The  arithmetical  mean 
of  the  responses  to  these  questions  in  each  case  favored  a positive  response. 
The  first  question  means  were  each  halfway  between  "moderate  value"  and 
"great  value"  on  the  scale.  A statistical  test  at  the  95  percent  confidence 
level  demonstrated  no  significant  difference  between  the  two  sample  means. 

The  second  question,  which  addressed  "the  degree  IGT&E  recommendations 
are  followed,"  provided  very  similar  results  with  no  significant  dif- 
ferences in  the  means  observed.  The  sample  response  means  were  again  ap- 
proximately halfway  between  "moderately"  and  "greatly."  A look  at  the 
remaining  questions  and  their  responses  reveals  only  "good"  or  positive 
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opinion  trends  on  all  responses  that  indicated  a group  opinion  (see  Table  2). 
This  holds  true  even  when  considering  only  an  80  percent  confidence  level 
for  a group  opinion.  The  80  percent  confidence  level  was  also  checked, 
since  many  responses  indicated  no  opinion  at  the  90  percent  confidence 
level.  In  presenting  this  type  data,  I realize  some  of  the  potential 

_ » 

shortcomings,  since  the  sample  size  is  small,  and  the  different  question 
^ interpretation  by  each  individual  submitting  an  opinion.  The  response  to 

one  question  in  particular  leads  me  to  believe  the  two  groups  interpreted 
it  differently.  This  was  the  question  concerning  the  "opportunity  to  com- 
ment on  I OT&E  test  objectives  before  they  are  formalized."  "Formalized" 
to  the  I OT&E  project  manager  means  before  they  appear  in  a TAC-approved 
test  order  document.  "Formalized"  to  the  AFSC  project  manager  could  have 
meant  appearance  in  the  I OT&E  test  plan  which  follows  the  test  order.  From 
past  experience,  I know  that  the  detailed  test  plan  is  coordinated  much 
more  intensively  with  the  responsible  DT&E  project  manager  than  the  TAC 
project  order  which  lists  the  project  objectives  and  provides  TAl-JC  with 
the  authority  to  proceed  with  the  evaluation.  This  explanation  is  rein- 
forced somewhat  by  the  AFSC  project  managers'  responses  to  the  question 
dealing  with  the  extent  of  I OT&E  project  manager  coordination  with  DT&E 
project  manager  during  1 0T & E planning  (AFSC  Question  #13).  An  AFSC  group 
opinion  did  exist  with  an  80  percent  confidence  level.  The  fact  that  no 
TAWC  opinion  existed  indicates  that  the  group  could  not  agree  that  they 
had  adequate  opportunity  to  comment  on  IOT&E  test  objectives  prior  to 
their  issuance  by  higher  headquarters . This  does  not  take  advantage  of 
experienced  IOTSE  project  managers'  expertise  and  does  not  appear  to  be 
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an  optimal  approach  for  achieving  workable  test  objectives.  Workable  test 
objectives  are  extremely  important,  since  the  entire  effort  must  be 
structured  around  these  points. 

Since  no  real  controversy  was  revealed  from  the  questionnaire  responses, 
there  is  no  point  in  continuing  a lengthy  discussion.  The  statistical  analysis 
results  are  summarized  in  Table  2.  The  questions  were  also  categorized  ac- 
cording to  question  type  with  a brief  summation  of  respondents'  reactions 
to  the  questions.  This  is  included  as  Table  3.  Again,  it  can  be  seen 
that  the  overwhelming  positive  group  opinions  established  were  toward  those 
questions  relating  specifically  to  the  quality  of  IOT&E. 
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Table  3 

Summary  of  Reactions 


QUESTION  TYPE 


GENERAL  REACTIONS 


Quality  of  IOT&E  (General) 

AFSC  Questions  5,  15,  16,  18 
TAWC  Questions  5,  8,  9,  11, 
16,  20 


Only  one  group  response  to  these  ques- 
tions possessed  no  group  opinion.  This 
was  TAWC's  response  to  the  question  on 
duplication  of  other  reports  (No.  16). 
All  other  responses  indicated  definite 
opinions.  These  opinions  were  all  posi- 
tive. 


Coordination 

AFSC  Questions  8,  13,  14 
TAWC  Question  7 


AFSC  had  positive  group  opinions  on  two 
of  the  three  questions  relating  to  co- 
ordination. TAWC  had  none  on  their  one 
question  which  related  to  the  test  ob- 
jectives which,  in  my  opinion,  appears 
to  be  a significant  issue. 


Working  Relationship  with  Other 
Program  Managers 
AFSC  Question  11 
TAWC  Question  13 

Authority  of  IOT&E  Program 
Managers 

AFSC  Question  19 
TAWC  Question  20 


Again,  AFSC  program  managers  had  a 
definite  group  response,  indicating  a 
favorable  working  rel ationshi p--TAWC 
program  managers  had  no  group  opinion. 

There  was  general  positive  agreement 
between  both  groups  on  this  point. 
Apparently,  no  problem  exists  here. 


Realistic  Test  Environment 
TAWC  Question  15 


No  group  opinion  existed  as  to  the 
realism  of  test  environment. 


Logistics  Considerations  No  group  ooinion  existed  for  either 

in  IOT&E  TAWC  or  AFSC. 

AFSC  Question  17 
TAWC  Question  19 


Cost  Factors 

TAWC  Question  13 

Ad eouacy  of  Time  Allowed  for 
IOT&E  P Ian  and  Report 
TAWC  Questions  10,  21 

IOT&E  Effects  on  DT&E 

AFSC  Questions  7,  9,  10 


The  definite  group  opinion  here  was  ttat 
cost  caused  no  problems  with  the  IOT&E. 

No  group  opinion  exhibited. 


Only  one  group  opinion  existed  here. 

It  established  that  no  DT&E  schedule 
degradation  vas  experienced  as  a result 
of  the  IOT&E. 
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D.  AIR  FORCE  TEST  & EVALUATION  CENTER  (AFTEC) 

After  the  issuance  of  Mr.  Packard's  memoranda  in  1971  and  the  result- 
ing Air  Force's  restructuring  of  its  OT&E  policies,  some  criticism  of  its 
OT&E  activities  continued.  The  December  1972  Report  of  the  Commission  on 
Government  Procurement  states: 

To  be  independent  and  impartial,  OT&E  components  should 
be  distinctly  separate  from  the  organizations  that  sponsor  or 
will  use  a major  system. 

The  developmental  activity  should  not  be  the  evaluator 
of  his  own  product  . . . . It  is  very  difficult  for  those 
responsible  for  developing  a system  to  remain  objective 
about  it.  Therefore,  the  test  data,  evaluation,  and  report- 
ing must  be  credible  and  come  from  a separate  source. 

There  are  several  reasons  why  the  user  should  not  have 
primary  responsibility  for  operational  testing.  The  user 
is  directly  involved  on  the  evaluation  of  and  requirement 
for  the  system  and  usually  applies  pressure  to  have  the 
system  deoloyed  as  scon  as  possible.  The  user  has  many  over- 
riding demands  on  its  own  resources  that  are  concerned  with 
its  primary  missions.  Operational  testing  should  receive 
emphasis  and  become  a professional  career  activity  in  the 
agency.  A high  level  of  training  is  required  and  the  per- 
sonnel assigned  should  have  scientific,  operational,  and 
analytic  ski  11s.  During  the  past  cccacc , only  the  Air  Force 
has  assigned  operational  testi ng  to  the  user. 

The  organizational  arrangement  that  best  satisfies  the 
two  needs  described  above  would  be  the  one  similar  to  the 
Navy:  a_  separate  activity  reporting  to  a_  hi gh  level . The 

Army  recently  established  a separate  activity  to  perform 
OT&E.  (13:164) 

In  addition,  a Comptroller  General  Report  of  .“arch  26,  1973,  recommended: 

Continue  to  emphasize  operational  test  and  evaluation 
by  establishing  in  each  military  department  an  organization 
independent  of  the  developer  and  user.  The  senior  OSD  of- 
ficial in  this  activity  should  report  to  the  Secretary  of 
Defense  or  to  his  Deputy.  (14:54) 

In  essence,  the  Air  Force  has  been  the  only  remaining  Service  to  not  es- 
tablish an  independent  test  and  evaluation  organization.  This  situation 
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has  now  been  changed,  since  Air  Force  Regulation  23-36,  issued  in  January 
1974,  established  the  Air  Force  Test  and  Evaluation  Center  (AFTEC)  as  a 
separate  operating  agency.  The  mission  of  AFTEC  is  defined  in  the  regu- 
lation as  follows: 

AFTEC 's  mission  is  to  manage  the  Air  Force's  Operational 
Test  & Evaluation  program.  AFTEC  assesses  the  military  utili- 
ty of  major  and  HQ^  USAF  designated  non-major  Air  Force  systems. 

Major  commands  (MAJCOMS)  support  AFTEC  as  directed  by  HQ  USAF 
program  management  directives  (PMDs)  or  Test  Directives  (TDs). 

AFTEC  recommends  Air  Force  OT&E  policy  to  HQ  USAF  for  approval 
as  well  as  plans,  directs,  controls,  evaluates  and  reports-- 
independently--on  OT&E.  AFTEC  serves  as  the  principal  field 
command  for  providing  OT&E  information  to  the  Secretary  of  the 
Air  Force  (SAF)  and  the  Chief  of  Staff  of  the  Air  Force  (CSAF) 
in  preparation  for  OSARC  actions  and  procurement  requests  for 
which  OT&E  information  is  statutorily  required  to  be  supplied 
to  the  Congress. 

This  regulation  goes  on  to  define  other  mission  policies,  such  as: 

HQ  USAF  designates  OT&E  projects  that  AFTEC  will  con- 
duct as  well  as  some  that  MAJCOMS  will  conduct.  Other  OT&E 
projects  may  be  conducted  by  MAJCOMS  when  directed  bv  their 
respective  commanders.  (4:1) 

In  other  words,  AFTEC  will  be  primarily  responsible  for  OT&E  of  major  sys- 
tems with  major  command  OT&E  continuing  much  the  way  it  is  now.  It  is  en- 
visioned that  TAC's  testing  center,  the  Tactical  Air  Warfare  Center  (TAWC), 
will  continue  its  functions  of  OT&E  in  support  of  both  HQ  USAF  and  TAC- 
directed  OT&E.  For  further  clarification,  HQ  USAF  Program  Management  Di- 
rective (PMD)  74-1  categorizes  OT&E  into  four  groups  and  lists  responsi- 
bility as  follows:  (15) 
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AFTEC-CONDUCTED  OT&E  PROJECTS 


AF/XOOW 

AFTEC 

COMMANDS 

Draft  PHD 

OT&E  Input 

(Critical 

Issues)  (Test  Schedule) 

(Resource  Estimate) 

Final  PMD 

Draft  Test  Design 

Inputs 

(Initial  Test 

Project  Order 

Directive) 

Final  Test  Design 

Comments 

Form  Test  Team 
Preliminary  Test  Plan 

Coordi nate 

Obtain  Air 

Staff  Draft  Test  Plan 

Info 

Approval 

AFTEC/CC  Estimate 
Draft  Test  Directive 

Final  Test 

Directive  Final  Test  Plan 

Project  Plan 

Info 

Interim  Test  Reports 

Info 

Final  Test  Report 

Follow-up 

AFTEC/CC  Summary 
Follow-up 

Follow-up 

AFTEC -CONDUCTED  JOINT  OTSE  PROJECTS 

DDR&E 

AF/XOOW 

AFTEC 

COMMANDS 

Memo 

Initial  Test 
Directive 

Project  Order 

Obtain  Air  Staff 
Approval 

Designate  Test 
Di rector 

Approve 

Negotiate  Charter 

Info 

Form  Test  Team 

Info 

Manning 

Test  Design 

Obtain  Approval 

Negotiate 

Comment 

Approve 

Obtain  Approval 

Draft  Test  Plan  Coordinate 
AFTEC/CC  Estimate 
Draft  Test 
Directive 

Final  Test 
Directive 

Final  Test 
Plan 

Project  Plan 

?1 


COMMAND-CONDUCTED  OT&E  PROJECTS 


AF/XOOW 

Draft  PMD 
(Critical  Issues) 


Final  PMD 

(Final  Test  Directive) 


AFTEC 

Input 

(Test  Schedule) 
(Resource  Estimate) 

Info 

Approval 


COMMAND 


Project  Order 

Test  Design/ 
Test  Plan 


Info 


Follow-up 


Monitor 


Evaluate 

AFTEC/CC  Summary 
Follow-up 


Interim  Test 
Reports 

Final  Report 
Follow-up 


COMMAND-INITIATED  OT&E  PROJECTS 


AF/XOOW 


AFTEC 

Info 


Info 


Info 


COMMANDS 

Project  Order 

Test  Plan 

Project  Plan 

Interim  Test 
Reports 

Final  Report 

Follow-up 


While  the  rationale  for  creating  AFTEC  is  quite  clear,  the  reasoning  which 
allows  continued  MAJCOM  (User)  I OT&E  is  not.  The  same  rationale  that  ap- 
plies to  major  systems  appears  to  also  apply  equally  to  the  others. 
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E.  CONCLUSIONS  AND  RECOMMENDATIONS 

1.  Conclusions: 

Mr.  David  Packard's  revisions  of  test  and  evaluation  policies  in 
1971  have  had  far-reaching  effects  on  the  Air  Force's  test  philosophy. 

This  has  affected  the  test  concept  for  all  subsystems,  whether  they  are 
categorized  as  major  or  not.  It  is  refreshing  to  note  that  many  Systems 
Command  program  managers  recognize  the  importance  of  user  involvement 
early  in  the  system  acquisition  phase.  The  two-way  exchange  between  AFSC 
and  TAWC  as  afforded  by  combined  DT&E/IOT&E  appears  to  have  been  widely 
accepted.  This  improved  communication  link  will  no  doubt  reap  immeasurable 
benefits. 

It  is  also  concluded  that  TAC/TAWC  IOT&E  performance  has  been  ex- 
cellent. No  negative  comments  were  received  during  the  course  of  inter- 
views, although  I made  several  attempts  to  ferret  some  out.  A few  nega- 
tive comments  were  recorded  on  the  questionnaires,  but  these  were  minor  and 
were  not  indicative  of  any  perceivable  trend.  Statistical  data  analysis 
revealed  only  the  existence  of  positive  opinions  from  each  sample  group. 

2.  Recommendations: 

The  objectives  of  the  developer  and  user/tester  are  usually  quite 
different.  Therefore,  I recommend  more  emphasis  be  placed  on  combined 
DT&E/IOT&E  testing  in  order  to  resolve  earlier  the  differences  between  the 
developer  and  user.  I believe  this  should  be  done  whenever  possible  and  not 
be  reserved  only  for  situations  where  it  must  be  warranted  by  program  sched- 
ules or  fund  limitations  as  implied  by  AFR  80-14.  I also  recommend  that 
for  every  acquisition  program,  AFTEC  or  the  using  command  (whichever  is 
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appropriate)  designate  an  individual  from  the  OT&E  testing  organization  to 
maintain  close  liaison  with  the  developer.  This  individual  should  be 
designated  at  the  earliest  possible  date,  and,  if  possible,  should  be  the 
prospective  I0T&E  or  OT&E  project  manager.  This  early  participation  is  in 
consonance  with  current  DOD  policies  and  would  motivate  the  user/tester  to 
influence  the  design  while  it  is  in  a flexible  state. 
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ANNEX  A 


SAMPLE  QUESTIONNAIRE  AND  COVER  LETTER 
OF  THE  TYPE  MAILED  TO 


AFSC  PROGRAM  MANAGERS 


DEPARTMENT  OF  DEFENSE 

DEFENSE  SYSTEMS  MANAGEMENT  SCHOOL 
FORT  BELVOIR.  VIRGINIA  22060 


DSMS-PMC 

SUBJECT:  Student  Study  Paper  Questionnaire 


1.  My  name  is  Tom  Julian  and  I am  currently  a student  at  the  Defense 
Systems  Management  School  (DSMS),  Fort  Belvoir,  Virginia.  One  of  the 
academic  requirements  of  the  School  is  the  completion  of  a study  paper 
relating  to  a management  topic. 

2.  The  purpose  of  the  attached  questionnaire  is  to  collect  data  for  the 
study  paper,  which  will  be  titled  "A  Look  into  the  Effectiveness  of  TAC 
IOT&E".  One  purpose  of  the  study  is  to  gain  some  insight  into  how  wel  1 
the  relatively  new  concept  of  combined  DT&E/IOT&E  is  working  out.  The 
questionnaire  is  intended  for  a program  manager  who  has  actual  experience 
in  conducting  the  DT&E  portion  of  a combined  DT&E/IOT&E.  There  are  no 
right  answers  for  the  questionnaire.  I am  soliciting  your  opinions  and 
any  comments  you  would  like  to  make.  Anonymity  of  the  questionnai re  will 
be  maintained. 

3.  I am  under  a very  tight  academic  schedule,  so  a rapid  response  would 
be  appreciated.  Request  return  of  the  questionnaire  by  19  April  1974. 

4.  If  any  questions  arise  concerning  this  questionnaire  or  if  discussion 

is  desired,  I can  be  reached  at  the  DSMS,  AUTOVON:  354-5536. 

5.  Any  assistance  you  or  any  of  your  associates  can  provide  will  be 
greatly  appreciated. 


1 Inclosure 
Questionnaire 


THOMAS  H.  JULIAN,  DAFC 
Program  Management  Course  74-1 
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QUESTIONNAIRE 


NOTE:  If  you  have  had  management  experience  on  more  than  one  combined 

DT&E/IOT&E,  please  base  your  opinions  on  the  last  one  completed. 

Answers  to  the  scaled  questions  should  be  indicated  by  a mark  with- 
in one  of  the  five  spaces  provided. 

1.  How  did  you  initially  become  familiar  with  the  general  intent  of 

IOT&E  as  defined  in  DOD  Directive  5000.1  or  AFR  80-14?  (Check  one  answer. 

a.  OJT/conversation  with  fellow  workers. 

b.  Command  information/correspondence. 

c.  Read  regulations/directives. 

d.  Briefed  by  supervisor. 

e.  Other  (please  specify).  


2.  How  many  joint  DT&E/IOT&E  programs  have  you  been  associated  with?  

3.  How  many  have  you  managed  to  completion?  

4.  During  what  calendar  year  was  your  last  test  completed?  

5.  In  your  opinion,  how  valuable  was  the  10T&E  portion  to  the  best  interests 
of  the  Air  Force? 


MODERATE  ' GREAT 

NO  VALUE  VALUE  VALUE 

Comments:  • 


6.  In  which  of  the  following  categories  would  you  classify  the  test  item 
at  the  time  of  I0T&E?  (Check  one  answer.) 

a.  Pre-production  prototype. 

b.  Pilot  production  item. 

c.  Breadboard  construction. 

d.  Commercial  off-the-shelf. 

e.  Other  (describe).  


7.  To  what  degree  did  you  allow  for  IOT&E  during  the  DT&E  planning? 


NONE 

MODERATELY 

GREATLY 

Comments : 


8.  How  much  opportunity  did  you  have  to  comment  on  proposed  IOT&E  test 
objectives  before  they  were  formalized? 


NONE  MODERATE  GREAT 

Comments : 


9.  To  what  degree  do  you  feel  that  accomplishment  of  DT&E  test  objectives 
was  effected  by  using  command  participation? 


NONE  MODERATE  GREAT 

Comments : 


10.  In  your  opinion,  to  what  degree  was  your  test  completion  schedule  ad- 
versely affected  by  the  inclusion  of  IOT&E? 


NONE  MODERATE  GREAT 

Comments:  


11.  How  would  you  rate  your  working  relationship  with  the  IOT&E  project 
manager? 


POOR 

AVERAGE 

EXCELLENT 

Comments : 
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12.  Would  you  have  preferred  to  conduct  your  test  independently,  without 
using  command  involvement? 

a.  YES 

b.  NO 

Comments : 


13.  To  what  degree  did  the  using  command  coordinate  with  you  during  test 
planning? 


NONE  M 

ODERATELY  GREATLY 

Comments : 


14.  To  what  degree  did  the  using  command  coordinate  with  you  during 
preparation  of  their  test  report? 


NONE 

MODERATELY 

GREATLY 

Comments : 


15.  To  what  extent,  in  your  opinion,  did  the  IOT&E  test  report  duplicate 
those  of  any  other  Service  or  agency? 


| 

I 

NONE 

Comments : 

MODERATELY 

GREATLY 
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16.  In  your  opinion,  how  would  you  rate  the  IOT&E  conclusions  and  recom- 
mendations? 


COMPLETELY 

INADEQUATE 


COMPLETELY 

ADEQUATE 


Comments : 


17.  In  your  opinion,  how  well  did  the  IOT&E  results  identify  logistic 
support  requirements? 


COMPLETELY 

INADEQUATELY 


COMPLETELY 

ADEQUATELY 


Comments: 


18.  To  what  degree  do  you  think  the  IOT&E  recommendations  will  be  cr  have 
been  followed? 


1 

NONE 

MODERATELY 

GREATLY 

Comments : 


19.  How  would  you  rate  the  adequacy  of  the  IOT&E  manager's  authority  to 
conduct  his  portion  of  the  evaluation? 


COMPLETELY 

INADEQUATE 


COMPLETELY 

ADEQUATE 


Comments : 


ANNEX  B 


SAMPLE  QUESTIONNAIRE  AND  COVER  LETTER 
OF  THE  TYPE  MAILED  TO 
TAWC  IOT&E  PROJECT  MANAGERS 


DSMS-PMC 


DEPARTMENT  OF  DEFENSE 
DEFENSE  SYSTEMS  MANAGEMENT  SCHOOL 
FORT  BELVOIR.  VIRGINIA  22060 


SUBJECT:  Student  Study  Paper  Questionnaire 


1.  My  name  is  Tom  Julian  and  I am  currently  a student  at  the  Defense 
Systems  Management  School  (DSMS),  Fort  Belvoir,  Virginia.  One  of  the 
academic  requirements  of  the  School  is  the  completion  of  a study  paper 
relating  to  a management  topic. 

2.  The  purpose  of  the  attached  questionnaire  is  to  collect  data  for 
the  study  paper,  which  will  be  titled  "A  Look  into  the  Effectiveness 
of  TAC  IOT&E".  One  purpose  of  the  study  is  to  gain  some  insight  into 
how  well  the  relatively  new  concept  of  combined  DTuE/IOT&E  is  working 
out.  The  questionnaire  is  intended  for  a project  manager  who  has  actual 
experience  in  conducting  the  IJTSE  portion  of  a combined  DT&E/IOTSE. 

There  are  no  right  answers  for  the  questionnaire.  I am  soliciting  your 
opinions  and  any  comments  you  would  like  to  make.  Anonymity  of  the 
questionnaire  will  be  maintained. 

3.  I am  under  a very  tight  academic  schedule,  so  a rapid  response  would 
be  appreciated.  Request  return  of  the  questionnaire  by  19  April  1974. 

4.  If  any  questions  arise  concerning  this  questionnaire  or  if  discussion 

is  desired,  I can  be  reached  at  the  DSMS,  AUTOVON:  354-5536. 

5.  Any  assistance  you  or  any  of  your  associates  can  provide  will  be 
greatly  appreciated. 


1 Inclosure 
Questionnai re 


THOMAS  H.  JULIAN,  DAFC 
Program  Management  Course  74-1 


QUESTIONNAIRE 


NOTE:  If  you  have  had  management  experience  on  more  than  one  combined 

DT&E/IOT&E,  please  base  your  opinions  on  the  last  one  completed. 

Answers  to  the  scaled  questions  should  be  indicated  by  a mark  with- 
in one  of  the  five  spaces  provided. 

1.  How  did  you  initially  become  familiar  with  the  general  intent  of 

IOT&E  as  defined  in  DOD  Directive  5000.1  or  AFR  80-14?  (Check  one  answer. 

a.  OJT/conversation  with  fellow  workers. 

b.  Command  information/correspondence. 

c.  Read  regulations/directives. 

d.  Briefed  by  supervisor. 

e.  Other  (please  specify).  


2.  How  many  joint  DTSE/IOT&E  programs  have  you  been  associated  with?  

3.  How  many  have  you  managed  to  completion?  

4.  During  what  calendar  year  was  your  last  test  completed?  

5.  In  your  opinion,  how  valuable  was  the  IOT&E  portion  to  the  best  interests 
of  the  Air  Force? 


MODERATE 

NO  VALUE  VALUE 

Comments : 


6.  In  which  of  the  following  categories  would  you  classify  the  test  item 
at  the  time  of  IOT&E ? (Check  one  answer.) 


a . 

Pre-production  prototype. 

b. 

Pilot  production  item. 

c. 

Breadboard  construction. 

d. 

Commercial  off-the-shelf. 

e. 

Other  (describe).  

GREAT 

VALUE 
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7.  How  much  opportunity  did  you  have  to  comment  on  proposed  IOT&E  test 
objectives  before  they  were  formalized? 


8.  In  your  opinion,  how  would  you  compare  your  particular  overall  test 
purpose  with  one  which  had  been  designed  to  facilitate  a major  program 
decision? 


COMPARISON  COMPARISON  IDENTICAL 

Briefly  describe  the  decision  in  your  particular  case.  


9.  In  your  opinion,  to  what  degree  were  your  test  objectives  adequate  to 
enable  a valid  test  and  subsequent  HQ  decisions? 


10.  How  would  you  rate  the  time  available  to  plan  the  test? 


COMPLETELY 


COMPLETELY 


INADEQUATE 


ADEQUATE 


Comments: 


11. 


To  what  extent  were  you  able  to  collect  adequate  data  to  satisfy  all 
test  objectives? 


COMPLETELY 
ADEQUATE 

Comments: 


COMPLETELY 

INADEQUATE 


12.  If  your  answer  to  the  above  question  favored  an  inadequate  rating 
(i.e.,  left  of  central  point),  to  what  would  you  attribute  the  major 
source  of  difficulty?  (Check  as  many  as  appropriate.) 

a.  Insufficient  time. 

b.  Poorly  stated  test  objectives. 

c.  Inadequate  test  objectives. 

d.  Unrealistic  planning. 

e.  Poor  AFSC  cooperation. 

f.  Priority. 

g.  Insufficient  support. 

h.  Other  (describe  briefly).  


13.  How  would  you  rate  your  working  relationship  with  the  AFSC  program 
manager? 


1 

J 

- 1 

POOR  AVERAGE  EXCELLENT 

Comments : 


14.  Would  you  have  preferred  to  conduct  this  test  as  a separate  IOT&E 
independent  of  AFSC? 

a.  YES 

b.  NO 

Comments:  
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15.  To  what  degree  do  you  feel  your  test  environment  simulated  an  actual 
operational  situation? 


COMPLETE 

DISSIMILARITY 

Comments : 


COMPLETE 

SIMILARITY 


16.  To  what  extent,  in  your  opinion,  did  your  test  report  duplicate  those 
of  any  other  Service  or  agency? 


NONE 

Comments : 


MODERATE 

DUPLICATION 


GREATLY 


17.  In  your  opinion,  to  what  degree  were  you  restricted  by  political  or 
other  factors  from  making  what  you  considered  the  most  valid  recom- 
mendations and  conclusions? 


NONE 

Comments : 


MODERATELY 

RESTRICTED 


GREATLY 

RESTRICTED 


18.  To  what  degree  did  test  cost  factors  restrict  your  evaluation? 


NONE 

MODERATE 

GREAT 

Comments : 


19.  In  your  opinion,  how  well  did  the  IOT&E  results  identify  logistic 
support  requirements? 


INADEQUATE  ADEQUATE 

Comments : 


20.  To  what  degree  do  you  think  your  recommendations  will  be  or  have  been 
followed? 


21.  How  would  you  rate  the  adequacy  of  time  allowed  to  properly  write  and 
publish  the  report? 


COMPLETELY  COMPLETELY 

INADEQUATE  ADEQUATE 


Comments : 


22.  In  your  opinion,  how  would  you  rate  the  adequacy  of  your  authority 
during  the  evaluation? 


COMPLETELY  COMPLETELY 

INADEQUATE  ADEQUATE 


Comments: 


23.  Any  additional  comments  may  be  included  in  the  space  below. 


ANNEX  C 


This  annex  contains  a complete  listing  of  all  scaled  questions  posed 
to  the  AFSC  and  TAWC  program  managers.  These  questions  are  grouped  with 
the  first  ones  being  those  common  to  both  groups  followed  by  only  the 
questions  addressed  to  AFSC  and  TAWC,  in  that  order.  The  responses  to 
each  question  are  shown  as  bar-graphs.  Included  also  is  a summarization 
of  all  comments  pertaining  to  the  particular  question.  General  comments 
not  related  to  any  one  particular  question  are  also  categorized  and  in- 
cluded at  the  rear  of  this  annex. 


The  following  listed  questions  are  those  which  were  common  to  both  the 
AFSC  and  TAWC  questionnaires: 

OUESTION:  In  your  opinion,  how  valuable  was  the  IOT&E  portion  to  the 

best  interests  of  the  Air  Force: 


AFSC  (Question  #5) 


TAWC  (Question  #5) 


8 


No  Great 

Value  Value 


No 

Value 


6 


Great 

Value 


AFSC  Comments: 

The  combined  DT&E/IOT&E  permits  the  user  to  make  an  assessment  of  the 
system  earlier  than  the  OT&E  and  gets  the  user  on  board  the  program  early. 

Speeds  system  development. 

The  system  was  so  complicated  that  many  problems  remained,  but  TAC 
was  able  to  determine  basic  usability  anyway,  and  clear  the  flak. 

It  validated  the  operational  capability  of  the  equipment. 

Reduces  hassle  with  the  user  during  OT&E  deployment. 


TAWC  Comments: 

IOT&E  gives  the  operator  a chance  to  get  his  views  into  final  con- 
figuration. 

I don't  think  the  decisions  made,  based  on  the  report,  were  proper. 

Several  operational  changes  were  incorporated  prior  to  full-scale 
production  as  a result  of  the  IOT&E. 

The  IOT&E  was  the  DT&E;  i.e.,  the  DT&E  folks  used  the  IOT&E  for  data 
collection . 

In  manv  respects,  the  IOT&E  also  included  gaps  in  the  basic  DT&E. 


C-l 


QUESTION:  How  much  opportunity  did  you  have  to  comment  on  proposed  IOT&E 

test  objectives  before  they  were  formalized? 

AFSC  (Question  #8)  TAWC  (Question  #7) 

7 7 

5 I I I I 

1 

°._  l~ 

None  Great  None  Great 

AFSC  Comments: 

The  IOT&E  objectives  were  jointly  developed  and  coordinated  with 
AFSC/TAC . 

IOT&E  objectives  are  added  after  DT&E  has  been  formalized. 

Intercommand  requirements  and  objectives  were  coordinated  through  the 
program. 

We  commented,  but  most  objectives  were  ignored.  Some  were  accepted. 
Was  responsible  for  the  planning. 

Test  group  established;  plans  started  at  the  beginning  of  program. 

TAWC  Comments: 

Would  a new  item  perform  in  an  acceptable  manner  and  provide  an 
improved  capability  over  the  present  item? 

Decision  (apparently  was  made  on  emotional  basis;  not  test  results) 
was  to  discard  equipment. 

Operational  suitability  and  effectiveness. 

Had  to  meet  DSARC  1 1 A for  additional  pre-production  missiles  for 
continued  test.  US  Navy  presented  the  DSARC. 

Data  was  available  to  make  decision;  however,  I don't  believe  anybody 
used  it. 

Qualification  to  two  manufacturers'  items  for  competitive  bid  purpose. 

One  test  does  not  give  me  a basis  to  properly  answer  this. 

Fly-off  between  two  systc~s  for  F-4  and/or  Pylon  carriage  chaff/IR 
counter  measures. 


C-2 


QUESTION:  How  would  you  rate  your  working  realtionship  with  the  AFSC/TAWC 

program  manager? 

AFSC  (Question  #11)  TAWC  (Question  #13) 

10 


Poor  Excellent  Poor  Excellent 

AFSC  Comments: 


Full-time  user  rep  in  SPO. 

I was  the  IOT&E  manager. 

IOT&E  project  manager  also  served  as  one  of  the  DT&E  project  test 
pilots. 

I have  been  project  manager  of  both  the  operational  and  the  develop- 
ment side. 

TAWC  Comments: 

Working  relationships  with  the  AFSC  Test  Engineer  are  generally 
better  than  with  the  AFSC  Program  Manager. 

None  on  this  test  because  we  worked  directly  with  the  Navy. 

They  tend  to  hide  their  dirty  laundry. 

Didn't  work  together. 

Working  level  good.  Supervisory  level  is  a problem.  This  may  be 
unique  to  my  situation  at  Eglin  AFB,  Florida. 


r 


\ 


QUESTION:  How  would  you  rate  your  working  realtionship  with  the  AFSC/TAWC 

program  manager? 


AFSC  (Question  # 11 ) 


0 

Poor 


10 


1__ 

J 

Excellent 


TAWC  (Question  #13) 


5 

2 

Poor 


Excel  lent 


AFSC  Comments : 

Full-time  user  rep  in  SPO. 

I was  the  IOT&E  manager. 

IOT&E  project  manager  also  served  as  one  of  the  DT&E  project  test 
pilots. 

I have  been  project  manager  of  both  the  operational  and  the  develop- 
ment side. 

TAWC  Comments: 

Working  relationships  with  the  AFSC  Test  Engineer  are  generally 
better  than  with  the  AFSC  Program  Manager. 

None  on  this  test  because  we  worked  directly  with  the  Navy. 

They  tend  to  hide  their  dirty  laundry. 

Didn't  work  together. 

Working  level  good.  Supervisory  level  is  a problem.  This  may  be 
unique  to  my  situation  at  Eglin  AFB,  Florida. 

* j 
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J 


QUESTION:  Would  you  have  preferred  to  conduct  your  test  independently 

without  the  other  command  involvement? 


AFSC  (Question  #12) 


1 

Yes  No 

AFSC  Comments: 


14 


TAWC  (Question  =14) 
10 


5 


Yes  No 


I strongly  support  early  involvement  of  the  using  command  in  the  test/ 
evaluation  of  systems. 

Having  the  operating  command  personnel  on  board  at  worse  is  an  im- 
provement. 

By  far  better  having  him,  if  he's  trying  to  help  instead  of  pick. 

The  joint  I OTSE  resulted  in  expediting  the  achievement  of  development 
and  operational  objectives. 


TAWC  Comments: 


Less  harrassment. 

Separate  IOT&E  with  a decent  hardware  sample  size  to  meet  objectives. 

Engineering  discrepancies  were  speedily  resolved  by  AFSC  allowing 
TAWC  to  handle  only  operational  deficiencies. 

His  expertise  was  invaluable. 

DT&E  objectives  are  often  counter-productive  to  I0TSE  objectives  and 
vice-versa.  This  produces  conflicts  over  how  limited  assets  are  to  be  used 
and  missions  flown. 


C-4 


QUESTION:  To  what  extent  in  your  opinion  did  the  IOT&E  report  duplicate 

those  of  any  other  Service  or  agency? 


AFSC  (Question  #15) 


TAWC  (Question  #16) 


8 


4 

4 

3 

2 

4 

3 

[ ^ 

1 



1 i ■ 1 ■ ' 1 1 . - .1 1 » 

None  Greatly  None  Greatly 


AFS_C  Comments: 

The  equipment  tested  was  never  tested  by  another  Service  or  agency. 

Not  yet  complete,  but  pretty  independent,  as  planned. 

The  test  report  did  overlap  some  DT&E  test.  However,  it  is  extremely 
valuable  to  obtain  the  using  command  perspective  and  views  regarding  the 
test  results.  In  particular,  maintenance  and  operations  evaluation  was 
essential  to  convince  CSAF  to  approve  and  fund  the  production  modification 
which  follows  the  DT&E/ IOT&E . 

80  percent  of  IOT&E  duplicated  the  results  of  the  DT&E  report. 


TAWC  Comments: 

The  Navy  has  conducted  similar  tests. 

Other  agencies  used  my  data  to  write  their  reports. 

With  only  enough  assets  to  adequately  perform  a DT&E,  the  report  is 
very  similar  to  a DT&E  report. 

Some  functional  and  maintainability  objectives  were  duplicated. 

None  that  I an  aware  of,  although  limited  planning  time  makes  detailed 
research  difficult. 


C-5 


QUESTION:  In  your  opinion,  how  well  did  the  IOT&E  results  identify 

logistic  support  requirements? 

AFSC  (Question  #17)  TAWC  (Question  #19) 


4 


4 


5 


C 


1 


Completely  Completely 

Inadequate  Adequate 


4 


3 

3 3 3 

Completely  Completely 

Inadequate  Adequate 


AFSC  Comments: 

Too  often  the  using  command  disregards  development  costs  in  order  to 
get  just  what  they  want. 

Original  Maintenance  Concept  was  modified. 

Logistic  support  was  not  addressed  since  supportabi 1 i ty  of  the  major 
portion  of  the  airborne  equipment  had  already  been  determined. 

This  will  continue  to  be  an  R&D  program  deficiency  until  sufficient 
R&D  funds  are  provided  to  buy  Blueline  Tech  Manuals  very  early  in  the 

program. 

Built-in  problem  of  the  specific  program.  AGE  is  expensive,  why  build 
it  unless  this  system  is  good,  but  how  do  you  evaluate  logistics  without 
it? 


TAWC  Comments : 

Not  really  included  in  test  objectives. 

Identification  of  logistics  support  requirements  should  not  be  a part 
of  IOT&E,  rather,  they  should  be  developed  independently  by  AFLC. 

Few  tests  are  long  enough  to  gather  realistic  logistic  or  maintenance 
data.  Answers  to  these  objectives  ore  by  necessity  a whitewash. 

This  was  closely  monitored. 

This  system  was  comparatively  simple.  Logistic  support  identification 
procedures  are  inadequate. 

Relied  on  contractor  for  all  logistic  aspects. 
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OUESTION:  To  what  degree  to  you  think  the  IOT&E  recommendations  will  be 

or  have  been  followed? 


AFSC  (Question  #18)  TAMC  (Question  #20) 


00 

5 

2 

2 2 

3 

0 

0 

1 

0 

None  Greatly  None  Greatly 


AFSC  Comments : 

All  recommendations  have  been  acted  on. 

It  appears  that  sufficient  funds  have  been  or  will  be  made  available 
to  have  at  least  95%  of  user  recommendations  incorporated  in  production 
hardware. 

An  equipment  production  and  Class  V modification  were  approved  as  a 
result  of  the  IOT&E. 

TAWC  Comments : 

Final  reports  require  too  much  coordination--everybody  changes  actual 
results  to  satisfy  their  own  opinions. 

Bureaucracies  are  not  immune  to  miracles. 

Because  of  budgetary  reasons  and  a change  in  requirements,  a decision 
not  to  produce  the  hardware  being  tested  was  made  prior  to  the  end  of  the 

IOT&E. 

All  recommended  action  is  being  acted  upon. 
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QUESTION:  How  would  you  rate  the  adequacy  of  the  IOT&E  manager's  authority 

during  his  portion  of  the  evaluation? 


AFSC  (Question  #19) 


0 


1 


_C 


COMPLETELY 

INADEQUATE 


COMPLETELY 

ADEQUATE 


TAWC  (Question  #22) 


CZD 


i 


COMPLETELY 

INADEQUATE 


COMPLETELY 

ADEQUATE 


AFSC  Comments : 

Can't  make  important  decisions  without  high-level  approval --especially 
in  TAC. 

IOT&E  manager  controlled  total  effort. 

The  only  problem--in  some  cases  IOT&E  testing  is  put  off  until  last 
and  cut  short. 

TAWC  Comments: 

Some  interference  from  superiors  not  completely  familiar  with  the 
problems. 


Be  nice  to  everyone;  after  all,  they  are  doing  you  a favor  because 
this  is  by  direction  a non-interference  test.  And  don't  cry  in  public. 

Project  manager  is  a form  pusher. 

A better  question  would  be,  "How  much  have  you  damaged  your  career 
by  making  decisions  which  those  higher  up  either  disagreed  with  or  felt 
were  beyond  your  limits  of  authority?" 
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AFSC  QUESTIONS 

The  following  questions  were  included  only  in  the  questionnaire  sent  to 
AFSC  program  managers. 

QUESTION:  To  what  degree  did  you  allow  for  IOT&E  during  the  DT&E  planning? 

AFSC  (Question  #7) 


J 

2 2 

None  Greatly 

AFSC  Comments: 

User  involved  since  conceptual  stage. 

The  program  was  a directed  joint  effort. 

DT&E/IOT&E  approach  began  with  initiation  of  the  program. 

The  operational  command  often  is  not  ready  for  planning  at  the  proper 

time. 


The  IOT&E  was  directed  by  CSAF/RDR  after  planning  was  completed  for 
DT&E.  The  DT&E/ IOT&E  planning  was  later  accomplished  and  the  IOT&E  ob- 
jectives were  developed  and  integrated  into  the  test  program. 


QUESTION:  To  what  degree  do  you  feel  that  accomplishment  of  DT&E  test 
objectives  was  effected  by  using  command  participation? 

AFSC  Response  (Question  #9) 


None 


Moderate 


Great 


AFSC  Comments: 

Favorably. 

The  joint  effort  eliminated  problems  that  might  have  been  otherwise 
generated. 

Mostly  technical  objectives  in  DT&E. 

With  qualified  flight  test  aircrews  rather  than  operational  aircrews, 
greater  system  accuracies  might  have  been  achieved. 

We  established  a TAC  deputy  test  director  and  obtained  outstanding 
support  from  the  user. 


QUESTION:  In  your  opinion,  to  what  degree  was  your  test  completion  sched- 

ule adversely  affected  by  the  inclusion  of  IOT&E? 


AFSC  Response  (Question  #10) 
9 


__3 

1 1 1 

r~  i i — > 

None  Moderate Great 

AFSC  Comments: 

As  a joint  effort--IOT&E  was  scheduled  along  with  the  later  phased 

DT&E. 

Delayed  the  planning  stage. 

No  effect  on  two-seaters.  Only  minor,  about  10'J-15",  on  single-place 
aircraft. 

The  drive  to  ready  the  aircraft  for  IOT&E  speeded  DT&E  somewhat. 

Not  affected  since  joint  effort  was  planned  at  start  of  program. 


QUESTION:  To  what  degree  did  the  using  command  coordinate  with  you  during 

test  planning? 

AFSC  Response  (Question  #13) 

7 


3 3 

2 I 


_0 I I 1 I 

None  Moderately  Greatly 

AFSC  Comments: 

Was  directed  joint  effort. 

Using  command  participated  in  test  planning  from  onset  of  program. 
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QUESTION:  To  what  degree  did  the  using  command  coordinate  with  you  during 

preparation  of  their  test  report? 


AFSC  Response  (Question  #14) 

6 

3 

1 

None  Moderately  Greatly 


3 

I 2 


AFSC  Comments: 

Draft  test  report  was  supplied  for  info  purposes. 

There  is  sometimes  a feeling  that  the  developing  command  wants  to 
sway  the  IOT&E  report. 

We  encouraged  TAC  to  evaluate  the  system  and  prepare  the  final  test 
report  independently. 

A post  test  meeting  was  held  to  review  test  and  insure  uniform 
interpretation  of  test  data. 
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QUESTION:  In  your  opinion,  how  would  you  rate  the  IOT&E  conclusions  and 

recommendations? 

AFSC  Response  (Question  #16) 

7 

5 

3 

* I 

, 0 , 0 

Completely  Completely 

Inadequate  Adequate 

j AFSC  Comments: 

Funds  shortage  limited  the  IOT&E. 

Premature,  but  probably  completely  adequate. 

Reports  of  the  operating  and  implementing  commands  were  objective 
and  generally  in  agreement. 


The  following  questions  were  included  only  in  the  questionnaire  sent  to 
ICT&E  project  managers. 

QUESTION:  In  your  opinion,  how  would  you  compare  your  particular  overall 

test  purpose  with  one  which  had  been  designed  to  facilitate  a 
major  program  decision? 


TAWC  Response  (Question  #8) 


2 

3 3 

1 

No  Comparison  Moderate  Comparison  Identical 


TAVIC  Comments: 

Identify  capabil ities,  limitations,  deficiencies  to  facilitate 
development  efforts. 

Would  a new  item  perform  in  an  acceptable  manner  and  provide  an 
improved  capability  over  the  present  item? 

Decision  (apparently  was  made  on  emotional  basis;  not  test  results) 
was  to  discard  equipment. 

Operational  suitability  and  effectiveness. 

Had  to  meet  DSARC  II  for  additional  pre-production  missiles  for  con- 
tinued test.  US  Navy  presented  the  DSARC. 

Qualification  of  two  manufacturer's  items  for  competitive  bids 
purpose. 

One  test  does  not  give  me  a basis  to  properly  answer  this. 

Fly-off  between  two  systems  for  F-4  and/or  Pylon  carriage  chaff/ 

IR  counter  measures. 
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QUESTION:  In  your  opinion,  to  what  degree  were  your  test  objectives 

adequate  to  enable  a valid  test  and  subsequent  HQ  decisions? 


TAWC  Response  (Question  #9) 


0 

Completely 

Inadequate 


Completely 

Adequate 


TAWC  Comments: 

Method  was  invalid  for  Tactical  evaluation;  test  plan  was  completely 
revised. 

Does  anyone  read  the  reports?  I had  objectives;  ASD  had  none. 

The  objectives,  again,  are  general;  the  testing  results  sell  the 
project. 

Objectives  too  sophisticated  for  available  hardware,  forcing  much 
analysis . 
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QUESTION:  How  would  you  rate  the  time  available  to  plan  the  test? 

TAWC  Response  (Question  #10) 


5 


4 4 

3 

1 

Completely Completely 
Inadequate  Adequate 


TAWC  Comments: 

If  aircraft  Class  II  Mod's,  etc.,  included  as  part  of  "planning,"  I 
just  barely  squeezed  by. 

Coordination  cycles  take  too  long  and  very  little  guidance  is  available 
prior  to  the  test  to  allow  pre-planning. 

TAC  is  usually  several  steps  behind  the  developing  command  and  is 
usually  late  with  the  project  order;  e.g.,  I received  P.0,  ten  days  after 
ADTC  began  DT&E/IQT&E. 

Adequate  planning  makes  actual  testing  easy. 

However,  it  was  done  during  prime  time. 

Normally,  we  have  2-3  weeks;  in  some  cases,  we  have  less. 
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QUESTION:  To  what  extent  were  you  able  to  collect  adequate  data  to  satisfy 

all  test  objectives? 


TAWC  Response  (Question  #11) 

7 


TAWC  Comments: 

Duration  of  IQT&E  does  not  lend  itself  to  realistic  maint.  and  failure 
data  collection. 

The  time  allowed  to  write  and  coordinate  the  final  report  is  much  too 
restrictive. 

Navy  test  and  Air  Force  was  dependent  on  then  for  all  data. 

By  hook  or  crook. 

One  test  objective  was  dropped  because  of  CSAF  time  constraint. 

We  do  good  work. 


QUESTION:  To  what  degree  do  you  feel  your  test  environment  simulated  an 


actual  operational  situation? 
TAWC  Response  (Question  #15) 


Complete  Complete 

Similarity  Dissimilarity 


TAWC  Comments: 

Sometimes  equipment  operators  are  too  well  trained. 

Not  ail  required  "real  world"  simulations  available. 

Depending  on  the  test,  varied  from  complete  dissimilarity  to  average 
similarity. 

The  test  facility  is  pririly  a one-on-one  system.  We  need  a multiple 
environment.  The  test  results  predict  single  engagement  probability  of 


QUESTION:  In  your  opinion,  to  what  degree  were  you  restricted  by  political 

or  other  factors  from  making  what  you  considered  the  most  valid 
recommendations  and  conclusions? 


TAWC  Response  (Question  #17) 

11 


TAV.'C  Comments: 

Higher  HQ  approval  stipulated  changes  in  some  conclusions. 
Item  was  tied  to  other  programs. 

Sample  size  too  restrictive  for  adequate  IOT&E  test  phase. 
It's  hard  to  tell  it  like  it  is. 

It  takes  considerable  effort. 
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QUESTION : To  what  degree  did  test  cost  factors  restrict  your  evaluation? 

TAWC  Response  (Question  #18) 

_8 

4 

3 

2 

h:  i ° j 

None  Moderate  Great 

TAWC  Comments: 

Tine  was  the  biggest  factor. 

Priority  and  overtime  money--definite  problem. 

Navy  paid  for  all  test  support.  AF  funded  TDY , only. 

Only  so  far  as  having  available  test  assets. 
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r 


1 


QUESTION:  How  would  you  rate  the  adequacy  of  time  allowed  to  properly 

write  and  publish  the  report? 

TAWC  Response  (Question  #21) 


7 


3 

3 

3 

o ( 

Complete! 

y 

Completely 

Inadequate  Adequate 


TAWC  Comments: 

Quality  and  completeness  are  the  sacrificial  lambs  of  unrealistic 
suspenses . 

Format  and  dates  primary  thing--we're  cutting  on  final  product. 

Generally  inadequate--this  was  an  unusual  case  because  timing  was 
based  on  availability  of  Navy  data--therefore , most  preliminary  work  was 
completed  prior  to  suspense  date  being  received. 

The  report  time  is  directed  by  individual  who  never  wrote  or  co- 
ordinated a report  and  the  following  formula  applies: 

timely  reports  = great  reports  = good  OER 

late  reports  = bad  reports 

This  grading  criteria  is  easy  to  understand,  simple  to  follow,  and  uniform 
throughout  the  command. 

Thirty  work  days--usual ly  fifteen  days  are  used  for  coordination  with- 
in TAWC. 

Difficult  to  do  justice  to  a complicated  test  within  the  thirty- 
working-day deadline. 
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SUMMATION  OF  QUALIFICATION  QUESTIONS 


QUESTION  1:  How  did  you  initially  become  familiar  with  the  general  intent 

of  IOT&E  as  defined  in  DOD  Directive  5000.1  or  AFR  80-14? 


ANSWER: 

AFSC 

TAWC 

2 

7 

a. 

OJT/conversation  with  fellow  workers 

3 

2 

b. 

Command  i nformati on/correspondence 

9 

11 

c. 

Read  regulations/directives 

0 

0 

d. 

Briefed  by  supervisor. 

2 

1 

e. 

Other. 

QUESTION  2:  How  many  joint  DT&E/IOT&E  programs  have  you  been  associated 

wi  th? 

ANSWER:  AFSC:  3.7  (average)  TAWC:  2.9  (average) 


QUESTION  3:  How  many  have  you  managed  to  completion? 

ANSWER:  AFSC:  2.5  (average)  TAWC:  1.7  (average) 


QUESTION  4:  During  what  calendar  year  was  your  last  test  completed? 

ANSWER:  AFSC:  73  (average)  TAWC:  73  (average) 


QUESTION  6:  In  which  of  the  following  categories  v/ould  you  classify  the  test 

item  at  the  time  of  IOTaE? 


ANSWER: 

AFSC 

6 

6 

0 

0 

4 


TAWC 


9 

a. 

Pre-production  prorotype 

8 

b. 

Pilot  production  item. 

1 

c. 

Breadboard  construction 

0 

d. 

Commercial  off-the-shelf 

0 

e. 

Other 
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AFSC  COMMENTS  INCLUDED  AT  END  OF  QUESTIONNAIRE 


I believe  the  support  ability  is  over-emphasized,  at  the  early  stages,  any- 
way. Some  systems  are  so  shakey,  their  utility  should  be  determined  before 
the  ribbons  are  tied.  Extensive  support  of  a useful  capability  is  more 
justified  than  good  support  of  a marginal  system.  "See  how  it  works  first!" 
That's  what  IOT&E  started  out  to  be. 


A.  All  questions  were  answered  using  following  base: 

1.  Program  tests  were  planned,  initiated,  and  partially  completed  prior 
to  80-14  revision  in  '72;  i.e.,  Cat  I,  II,  & III  tests  used,  rather 
than  DT&E , IOT&E,  OT&E. 

2.  Program  was  concurrent.  Production  decision  made  with  less  than 
10%  of  development  testing  accomplished. 

3.  Thus,  numerous  changes  in  hardware  and  software  were  made  throughout 
and  after  completion  of  DT&E  and  IOT&E. 


8.  I have  not  been  closely  associated  with  program  or  project  using  "new" 
concept  of  user  preparation  of  IOT&E  test  plan. 


The  concept  of  having  developing  and  using  agency  on  board  and  participating 
actively  from  program  conception  through  test  is  invaluable.  With  their 
pilots  flying  on  early  DTSE  sorties,  we  were  able  to  incorporate  changes 
early  during  DT&E  and  only  very  minor  changes  resulted  from  the  pure  IOT&E 
sorties. 


The  IOT&E  concept  is  a vast  improvement.  However,  test  personnel  require 
time  to  learn  how  to  use  the  system  and  will  make  it  work  much  better  in 
the  future. 


The  program  experience  related  in  this  questionnaire  was  the  first  Drone 
system  to  complete  a joint  AFSC/TAC  DT&E/IOT&E.  I am  convinced  that  user 
participation  in  the  DT&E/IOT&E  (as  compared  with  the  DT&E  followed  by  an 
OT&E)  provided  a more  thorough  evaluation  of  the  system  and  prepared  the 
user  to  plan  and  conduct  the  OT&E. 


TAWC  COMMENTS  AT  END  OF  QUESTIONNAIRE 


Two  major  deficiencies  exist  for  joint  IOT&E's: 

1.  By  the  time  test  objectives  are  received,  the  test  hardware  has 
been  bought  and  the  DT&E  has  been  planned.  This  results  in  either 
using  a lot  of  analysis  in  -meeting  objectives  or  attempting  to 
change-^rrr'existing  method  of  test,  often  during  active  testing. 

2.  When  AFSC  contracts  for  DT&E/IOT&E  hardware,  the  IOT&E  is  seeming- 
ly not  considered  for  the  production  decision.  A recent  example: 

DT&E--200  items 
I0T&E--  15  items 

In  actuality,  the  IOT&E  input  is  as  important,  if  not  more  important, 
than  the  DT&E  input. 

Separate  IOT&E  testing  with  adequate  assets  to  meet  the  IOT&E  objectives 
is  essential  for  an  objective  and  intelligent  IOT&E  input  for  a production 
decision.  Along  with  this,  reporting  should  include  sufficient  time  to 
present  the  results. 


The  test  discussed  here  was  the  Navy  Condor  Missile  Evaluation.  Total 
test  management  was  vested  in  the  Navy  with  AF  representatives  participat- 
ing to  make  a subjective  evaluation  of  the  weapon's  utility  to  the  AF 
role.  The  Navy  cooperated  very  well  with  the  AF  reps,  but  because  of  the 
numerous  differences  in  test  philosophy  between  the  two  Services,  some  of 
the  AF  time  schedules  had  to  be  adjusted. 


Independent  evaluation  is  the  best  way  to  conduct  DT&E/IOT&E.  The  develop- 
ing command  has  too  much  pride  of  authorship. 


Additional  time  required  to  write  final  reports.  This  is  end  product  of 
complete  contracting  and  testing  cycle  and  were  cutting  it  short  to  meet 
unrealistic  dates. 


Not  enough  people  support  assigned  to  IOT&E. 
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ANNEX  D 


SAMPLE  STATISTICAL  COMPUTATIONS 


The  following  is  a sample  of  calculations  used  in  the  statistical  anal- 
ysis. These  are  provided  only  as  an  example  of  the  method  used  and  are 
performed  using  the  responses  to  Question  =5. 

The  first  set  of  computations  was  made  to  determine  i f a group  opinion 
existed.  The  null  hypothesis  (Ho)  is  that  no  group  opinion  exists  (i.e., 
that  responses  are  distributed  evenly  for  all  possible  choices).  The 
Kilmogorov-Smirnov  method  is  used.  (Reference  Fundamental  Statistical 
Concepts  by  Fisher). 


AFSC  responses  for  Question  #5  were  as  follows: 


S' 


1 L 

No 

Value 


2 


8 


Great 

Value 


1 2 3 4 5 

An  even  cumulative  distribution  of  responses  would  be  as  follows  (i.e.,  no 
group  opinion): 


1 2 3 4 5 

5 5 5 5 5 

For  sample  size  of  15  yields: 

3_  _6_  _9_  12_  lj)__ 

15  15  15  15'  15 

Actual  responses  are: 

0 0 _5_  1 15_ 

15  15'  15 

The  maximum  difference:  Driax  = 6-0  = 6 

15'  15 

From  Table  K of  reference,  Dc  = .30  for  a sample  size  of  15,  and  0(  = .10, 
which  provides  a 90  percent  confidence  interval. 

If  Dmax  > Dc , reject  Ho,  indicating  that  a group  opinion  does  exist. 


Therefore,  reject  Ho. 

Dc  = .3 
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The  following  calculations  were  used  to  test  for  significant  difference  of 
the  two  sample  means  from  each  of  the  questions  common  to  both  groups  of 
program  managers.  Again,  these  are  done  (for  example  only)  using  Question  #5. 


The 

null 

hypothesis 

to  be  tested  is: 

Ho:  Xi  - X2  = 

= 0 

AFSC 

Response 

TAWC  Response 

Xi 

N 

(Xi  - N) 

(Xi-X-j)2 

N 

(Xi -N) 

(Xi-X2)2 

1 

0 

0 

10.24 

0 

0 

3.61 

2 

0 

0 

4.84 

1 

2 

4.41 

3 

5 

15 

1 .44 

2 

6 

1.21 

4 

2 

8 

.04 

8 

32 

.01 

5 

8 

40 

1.44 

6 

30 

1.21 

15" 

63 

18.00 

70 

16.45 

*1  = 

= 4.2 

X2  = 

= 4.1 

Ni  = 

= 15 

N2  = 

= 17 

The  pooled  estimate  of  the  common  variance  is  given  by: 


S2  = ^EjX:^)2  + ^(Xi-X2)2 

N]  + 2 


S2  = 18.00  + 16.45  = ! 

15+17-2 


S = p .148  = 1.07 

Referring  to  the  table  of  critical  values  of  t,  it  is  found  that  the  value 
of  t for  C<  = .05  (95%  confidence)  is:  t = 1.645 


Therefore,  the  hypothesis  Ho  will  be  rejected  for  calculated  t values  in 
excess  of  1.645.  Calculating  t provides: 


t = (X't  - X2)  = 4.2  - 4.1 


+J_ 

n2 


1.0| 


rr 

X5“ 


+ 1 

17“ 


t = .1  = .267 

rro7TT35l 
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1.645  > .267;  therefore,  the  hypothesis  that  no  significant  difference 
in  means  exists  is  accepted. 
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ABBREVIATIONS 


IOT&E 

Initial  Operational  Test  & Evaluation 

DOD 

Department  of  Defense 

AFTEC 

Air  Force  Test  and  Evaluation  Center 

TAC 

Tactical  Air  Command 

OSARC 

Defense  Systems  Acquisition  Review  Council 

JCS 

Joint  Chiefs  of  Staff 

BRDP 

Blue  Ribbon  Defense  Panel 

AFSC 

Air  Force  Systems  Command 

DT&E 

Development  Test  and  Evaluation 

TAWC 

Tactical  Air  Warfare  Center 

HQ 

Headquarters 

USAF 

United  States  Air  Force 

ASD 

Aeronautical  Systems  Division 

AFB 

Air  Force  Base 

OT&E 

Operational  Test  & Evaluation 

DDR&E 

Deputy  Director  of  Research  & Engineering 

